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Summary of Review 

A new report published by the Manhattan Institute for Edueation Poliey, “The Effeet of 
Speeial Edueation Vouehers on Publie Sehool Aehievement: Evidenee from Elorida’s 
MeKay Seholarship Program,” attempts to examine the eomplex issue of how eompetition 
introdueed through sehool vouehers affeets student outeomes in publie sehools. The possi- 
ble eontributions of this report, however, are outweighed by researeh design problems, fail- 
ure to take into aceount alternative explanations, and unsubstantiated assumptions about the 
direetion of possible seleetion bias. Together, these problems eall into question the findings 
and render the eonelusions drawn from those findings highly suspeet. 
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Review 



I. Introduction 

A new report released by the Manhattan 
Institute for Education Policy, “The Effect 
of Special Education Vouchers on Public 
School Achievement: Evidence from Elor- 
ida’s McKay Scholarship Program,” has 
received some attention in the press' and is 
likely to be cited by advocates of private 
school vouchers in the future. The report, 
written by Jay P. Greene and Marcus Win- 
ters, attempts to examine the complex issue 
of how competition introduced through 
school vouchers affects student outcomes in 
public schools. An important contribution 
of this report to the literature of voucher 
competition publications is its focus on stu- 
dents who are enrolled in public school spe- 
cial education programs. However, this con- 
tribution is outweighed by errors in methods 
and analysis. In particular, the report does 
not include a clear explanation (“specifica- 
tion”) of the statistical model chosen; the 
analysis fails to take into account alternative 
explanations; and it includes unsubstantiated 
assumptions about the direction of possible 
selection bias.^ 

Elorida’s McKay Scholarship Program is 
open to any student in the state who has 
been classified with a learning disability. 
These vouchers pay the lesser amount of full 
tuition at the chosen private school or the 
full amount that the public school would 
have received had the student enrolled. Ac- 
cordingly, for students who wish to enroll in 
high-tuition private schools, the difference 
would be the responsibility of the parents or 
guardians. Under Elorida’s normal funding 
system, public schools would receive differ- 
ent amounts of money depending on the 
nature of the disability; thus, the maximum 
voucher size varies with the type of disabil- 
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ity classification. According to the report, in 
2006-2007 McKay Scholarships ranged 
anywhere from $5,039 to $21,907, with an 
average of $7,206. As discussed below, this 
fact becomes important with regard to as- 
sumptions made by the authors in their dis- 
cussion of possible selection bias in their 
analysis."' 

Given the special education focus of the 
McKay Program, the report addresses some 
important issues concerning voucher-based 
school choice programs involving this very 
important group of students. However, there 
are important weaknesses in the analysis and 
interpretations of the data that undermine 
any practical use of the results and conclu- 
sions. 

Most troubling are fundamental problems 
with variable and model specifications (as 
explained below). In addition, the report 
includes critical unsubstantiated assump- 
tions that lead to unwarranted weight given 
to estimates from the analysis. The report 
does appropriately note the serious problems 
with selection bias in any analysis of this 
type, and it does acknowledge, in the techni- 
cal version of the report, that the authors’ 
methodology does not fully correct for it.^ 
Yet, despite this disclaimer about the study’s 
limitations, the report subsequently presents 
arguments suggesting that the authors can in 
fact anticipate the direction of possible bias 
in their analysis, an assertion that I chal- 
lenge later in this review. 

Without any tests or appropriate literature 
substantiating these assumptions, the report 
leads the reader down a path with a prede- 
termined conclusion: that vouchers have a 
positive competition effect. Elorida’s 
McKay Scholarships, the report tells us. 
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improve the educational outcomes of those 
special education students who stay in pub- 
lic schools, choosing not to use a voucher. 
The theory of action behind this conclusion 
is that increased competition to enroll these 
students leads public schools to improve the 
services or programs for the students not 
choosing to leave 

If valid, such conclusions have key policy 
implications. However, the Manhattan report 
inadequately addresses several critically 
important issues, discussed below, calling 
the report’s conclusions into serious ques- 
tion. 

II. Findings and Conclusions 
OF THE Report 

The scope of the report is narrowly focused 
and is very brief given the complexity of the 
analysis attempted. This brevity leaves many 
questions unanswered and makes it difficult 
to thoroughly examine the methods used or 
conclusions reached. However, the omis- 
sions themselves are evidence that the find- 
ings of this report should be viewed very 
cautiously and should not, without substan- 
tial confirmation and reanalysis, be used to 
make policy decisions regarding similar 
types of voucher programs. 

The report begins by outlining the critical 
questions at issue in the analysis. The nation 
has seen a vigorous debate about whether 
private school vouchers promote competi- 
tion between public and private schools, and 
more importantly whether that competition 
increases student outcomes for both groups 
of students (those who leave the public sec- 
tor for private schools and those who re- 
main). 

The authors suggest that their analysis is 
designed to directly address this question by 
providing estimates of the effect on public 



school productivity of offering private 
school vouchers to students with disabilities. 
In their report this design is actualized by 
examining whether those students that re- 
main have better schooling environments as 
a result of the public school response to the 
threat of losing students to private schools. 
The authors contend that Florida’s McKay 
Scholarship Program provides an excellent 
proving ground for this analysis because it is 
the largest private school voucher program 
in the nation and because it has seen a large 
increase in participation — rising to ap- 
proximately 4.5% of eligible students (in the 
2006-2007 school year, about 18,200 of 
nearly 400,000 students with disabilities in 
Florida). 

The authors’ main conclusion is that there is 
some evidence that suggests outcomes for 
students in public school special education 
programs improve with increased exposure to 
voucher opportunities.^ The authors esti- 
mated relatively small effect sizes of 0.05 
and 0.07 standard deviation units in mathe- 
matics and reading scores (respectively) for 
students with specific learning disabilities 
and average exposure to voucher-accepting 
private schools. (Exposure to private school 
vouchers is defined at the number of schools 
that accept vouchers within a 5- or 10-mile 
radius). In presenting these results, the au- 
thors assert that they are likely lower-bounds 
and that the actual benefits are likely greater 
since any selection bias that may exist is 
likely to bias the estimates downward. 

Yet, as discussed below, these estimates and 
this conclusion are based on poor model 
specifications, unclear analytic decisions, 
and questionable selection bias assumptions. 

III. The Report’s Use oe 
Research Literature 

In the authors’ very brief discussion, they 
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characterize the findings in the researeh lit- 
erature on eompetition in a relatively bal- 
aneed way. They suggest that there is eon- 
flieting evidenee on the question of whether 
publie sehool outeomes improve when ex- 
posed to greater eompetition, either from the 
private seetor or from publie seetor ehoiee 
alternatives sueh as eharter sehools. In addi- 
tion, they mention that several studies have 
found, using different methodologies, posi- 
tive outeomes of Florida’s aeeountability 
system, ineluding its voueher provisions. 

However, in a different seetion of this re- 
port, where the authors diseuss possible se- 
leetion bias in their analysis, they use sup- 
porting literature that, while somewhat ap- 
propriate, does not fully eharaeterize the 
unique issues faeed by students with dis- 
abilities in this poliey environment. This 
exelusive use of tangentially appropriate 
literature gives the superfieial but ineorreet 
impression that the explanations of possible 
bias raised in this literature are valid and 
eomplete. 

In addition, the report eompletely omits any 
literature about the testing of students with 
disabilities and how aeeommodations may 
affeet the results and outeomes of their 
study. The laek of substantive knowledge 
about the group examined may seriously 
eompromise the validity of the report’s find- 
ings. 

IV. Review of the Report’s 
Methodologies 

The main text of the technical report de- 
seribes the data used as the universe of pub- 
lie sehool data from Florida between the 
years of 2000-2001 and 2004-2005 (five 
years of data). However, within the analysis 
(speeifieally tables 3 and 4), there are eon- 
flieting numbers of years present. Table 3 
shows four years of data when deseribing 
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the number of private sehools aeeepting 
MeKay voueher, and Table 4 shows only 
three dummy variables (two presented and 
one omitted) suggesting only three years in 
the regression analysis. The year variables 

o 

are omitted from all other tables. 

In addition, the report’s key exposure vari- 
able (number of private sehools aeeepting 
vouehers within 5 and 10 miles of the 
sehool) is seriously flawed. In urban areas, 
multiple publie sehools likely share the 
same pool of voueher-aeeepting private 
sehools. A private sehool “competing” with 
three publie sehools is likely to have a 
weaker effeet (all other things being equal) 
on any given sehool than a private sehool 
eompeting with only one publie sehool. The 
supply of voueher vaeancies depends on 
both the number of spots available in the 
private sehools and the pool of potential 
publie sehool students near those sehools. 
This suggests that the number of private 
sehools willing to aeeept vouehers is less 
important than the number of available spots 
relative to the number of available publie 
sehool students who eould fill those spots. 
None of this is aeeounted for in any of the 
models estimated in this report. 

Future researehers engaging in sueh analy- 
ses may want to use, as a measure, the num- 
ber of spots available in the private seetor 
relative to the number of publie sehool spe- 
eial edueation students in similar grade lev- 
els within a ehosen distanee. Sueh a measure 
would be a mueh stronger indieator of the 
loeal supply of voueher spots available, 
sinee it would eompare aetual spots that 
eould be taken by students in that partieular 
sehool. 

Another eoneern is that the exposure meas- 
ure used in the report is actually measuring 
how urban the area surrounding the student 
is rather than the supply of voueher spots. 
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This confounding of the two variables (ur- 
banicity and exposure to vouchers) is due to 
the fact that urban schools would naturally 
have more public and private schools in 
close proximity to one another. Thus, any 
estimated effect attributed to the exposure 
variable would be partially due to the public 
school’s location in an urban area, relative to 
a rural or suburban public school. This con- 
found is particularly important because we 
know that urban schools generally perform 
worse than their suburban and many of their 
rural counterparts. 

Such a modeling problem could be easily 
addressed by including appropriate geo- 
graphical control variables (such as whether 
the area in whieh the student is attending 
school is urban, rural or suburban). This 
simple approach is not explored in the re- 
port, nor is any reason provided for this 
omission.^ 

These are the two most significant examples 
of the numerous vague descriptions and poor 
variable choices present in the report. Issues 
related to model specification and selection 
bias will be addressed in following sections. 

Achievement Analysis 

In the achievement regression models the 
report uses student scores on state standard- 
ized reading and mathematics tests as the 
outcome in an individual level, fixed-effects 
analysis''^ to control for unobserved individ- 
ual characteristics. It also includes a district 
fixed-effect variable, unspecified student 
characteristics, dummy variables indicating 
type of disability, the voucher exposure 
variable, and the interaction between disabil- 
ity type and exposure. The authors argue 
that the interaction between disability type 
and voucher exposure plus the main effect 
of voucher exposure represent the average 
effect of the McKay Program. 



There are several problems with the model. 
The report never states which individual 
characteristics were included in the analysis. 
Perhaps more importantly, the choice of a 
district fixed effect is curious given the hy- 
pothesis that school (not district) changes 
were responsible for improvements in stu- 
dent test scores. The district variable, par- 
ticularly in Florida (where countywide dis- 
tricts are the norm), simply does not make 
much sense as a control. 

In addition, within the text of the report the 
authors never clarify what test-score metric 
they use for their outcome. Florida reports 
both a developmental scale for their exami- 
nations as well as a criterion-referenced 
scaled score. The developmental score is 
useful for measuring changes year to year in 
an individual student; the criterion- 
referenced score is useful for comparing 
cohorts of students in the same grade from 
year to year. The appropriate measure to use 
here would be the developmental score, but 
again, the report does not state which is 
used.^^ 

Further, the sample in this analysis uses all 
grades (3-10) over all the years (presumably 
2000-2004). This choice of analytical frame- 
works virtually assures that there will be se- 
rious issues of attrition, since 10^’’ grade stu- 
dents in 2000 will only appear in the dataset 
one time, and 9* grade students in 2000 will 
appear only two times, and so on. This may 
account for the fact that the average number 
of observations for students is only about 2.5 
years for each of the achievement regres- 
sions, even though the dataset covers five 
school years. This choice may again lead to 
biases in the estimates; however, it is unclear 
which direction this bias would lead. 

Selection Bias 

The report includes a useful outline of vari- 
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ous types of selection bias that might be 
present in these estimates. However, the 
authors’ assumptions about the directionality 
and mechanisms driving the bias are ques- 
tionable, with many plausible mechanisms 
ignored and untested. 

Attrition/Choice Bias 

Non-random attrition from the sample is a 
critical problem for research such as this. 
The authors are attempting to determine 
whether the exit of students to private school 
(and the threat of that exit) affects the test 
scores of those who remain in public school. 
This begs question of whether a subsequent 
increase in the scores of public school spe- 
cial education students is simply due to low- 
scoring students exiting the public system 
with vouchers (which may upwardly bias 
estimates of the effectiveness of the McKay 
program). Alternatively, it is possible that 
students of higher ability exit with vouchers 
(which would downwardly bias the esti- 
mated McKay program effect). 

The authors suggest that Florida’s private 
schools are “creaming” the best students 
from the public school systems. Accord- 
ingly, their estimates of the McKay program 
effect would likely be a lower bound (un- 
derestimated). Unfortunately, the authors 
fail to consider factors other than “cream- 
ing.” For instance, the students who take the 
voucher are unlikely to be satisfied with the 
public system and may be performing at 
lower levels than their potential. One could 
also argue that relatively high-performing 
special education students would be less 
likely to transfer out of the schools in which 
they were performing better than their peers. 
Also, the federal No Child Left Behind Act 
(NCLB) provides an incentive for public 
schools to encourage lower-performing spe- 
cial education students to take advantage of 

12 

the voucher program and transfer out. 



Such transfers would have two positive 
main effects under NCLB for the public 
school: it could lower the number of special 
education students sufficiently to take that 
subgroup out of NCLB calculations (driving 
the number below the law’s subgroup report 
threshold), and it could leave behind higher- 
performing special education students, help- 
ing the school’s average subgroup score to 
meet Adequate Yearly Progress targets. 

Finally, since the severity of the disability is 

related to how much money comes with the 

voucher, there is a potential incentive for 

new private schools to open (or existing 

private schools to broaden their scope) and 

admit students with more severe disabilities, 

1 ^ 

and likely lower test scores. 

Each of these scenarios would result in an 
upward bias of the parameters estimated in 
the report (an overestimation of the voucher 
competition effect), rather than the down- 
ward bias that was conclusively posited in 
the Manhattan report. 

Assignment Bias 

The authors also discuss the possibility of 
bias in assignment of students to special 
education by public schools. They suggest 
that these schools might classify fewer stu- 
dents as eligible for special education since 
these schools would be reluctant to qualify 
students for the voucher if they were afraid 
of the competition. This, they argue, would 
lead to fewer students with mild disabilities 
being placed in special education, leading to 
an attenuation of the voucher effect. An al- 
ternative scenario for this assignment bias 
could be that schools are more likely to as- 
sign students that they wanted to leave the 
school such as those with behavior disor- 
ders. This may lead to the assignment of 
more students to special education, and 
those students may be relatively high- 
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performing academically (relative to other 
special education students), thus biasing the 
effect upwards. Raising such additional pos- 
sihilities is not intended as a criticism of 
those that the report includes; rather, the 
criticism is that the possibilities treated seri- 
ously in the report are only those that sup- 
port the conclusion that the results underes- 
timate the competition effect. 

Supply Bias 

As the report explains, private schools that 
accept vouchers could be making that deci- 
sion based on the type of nearby public 
schools. One type of supply bias would oc- 
cur if private schools were more likely to 
accept vouchers if located near public 
schools that are doing a relatively good job 
with their special education students. These 
schools could more effectively skim the 
cream. Alternatively, if private schools lo- 
cated near low-performing public schools 
were more likely to accept vouchers, they 
may be able to attract dissatisfied students. 
Again, the authors argue that the students 
transferring would be high-performing rela- 
tive to their peers. 

Both of these alternatives, argue the authors, 
would bias their estimates downward since 
more academically able students would be 
leaving, resulting in less able students re- 
maining in the public schools. This formula- 
tion of supply bias also relies on the premise 
that private schools skim the cream from the 
public system. As we have discussed in pre- 
vious sections, this contention is far from 
proven with regard to special education stu- 
dents. Private schools may have reasons for 
accepting lower-performing students, and 
the various motivations of those students 
and their families, as well as of the public 
schools and their employees, also play a 
complicated role. 



V. Review oe the Validity oe the 
Findings And Conclusions 

The report is on its most solid ground when 
it describes the challenges in performing an 
analysis such as this — an analysis that uses 
general administrative data and that does not 
include tracking information on the students 
who leave for the private system. 

However, the report’s findings rest on very 
weak foundations. The variables are vaguely 
defined and the models are poorly specified. 
The report also fails to take into account the 
possible effect of testing accommodations. 
The assumptions employed to explain the 
possible direction of selection bias is weak 
at best. And all of the conclusions rest on 
models that use a very weak measure of 
private school voucher supply. 

Moreover, consider the following two addi- 
tional concerns: 

1. The report does not sufficiently describe 
how such small numbers of students leaving 
public schools (an average of four per public 
school in Florida) would encourage such 
substantial changes in the behavior of public 
schools. Nor do the authors discuss how the 
mere presence of schools (absent large de- 
fections of special education students from 
the public schools) would trigger immediate 
changes in public school behavior that 
would be quickly reflected in student test 
scores. The report does not include a de- 
scription of how public school officials 
would know how many private schools in 
the local area were accepting vouchers or 
the level of capacity in these private schools 
to enroll additional students with disabili- 
ties. 

These issues of time lag and information 
gathering become important when one real- 



http://epicpolicv.org/thinktank/review-effect-of-special 



Page 6 of 9 



izes that the number of voueher recipients 
was quite modest until nearly halfway 
through the sampled time period. In order 
for the hypothesized competitive effects to 
have caused improvements in nearhy public 
schools, those schools would have had to 
almost immediately receive the signal that 
special education students were leaving their 
schools and then adjust their practices ac- 
cordingly, with the effects of these changes 
then very quickly having an impact on test 
scores. Such a series of events seems 
unlikely given, for instance, the difficulty 
schools are having meeting even the general 
testing expectations of NCLB. 

2. Also, the authors fail to account for the 
fact that special education students are ex- 
actly the group for which these standardized 
test scores have the least reliability, given 
the fact that, depending on the severity and 
type of disability, different accommodations 
are available to students. An important al- 
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ternative explanation for the authors’ find- 
ings could be that the longer special educa- 
tion students are in a school the better the 
school is at finding appropriate accommoda- 
tions — which would allow them to score 
better on the state standardized test. 

VI. Usefulness of the Report 
FOR Guidance of Policy 
AND Practice 

This report is a useful starting point for dis- 
cussions and research around school vouch- 
ers for students with disabilities. However, 
the analyses are so vague and the approach 
so flawed that their only practical use is as 
an initial template for addressing the impor- 
tant issues of selection bias for studies such 
as this. Any attempt to use this report for 
decision-making or policy evaluation, prior 
to validation using different methods and 
more robust approaches, should be viewed 
with extreme skepticism. 
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* Samuels, C. A. (2008). Vouchers a spur to public schools? Education Week, 27(6), 4. 

The Washington Times published a three-part op-ed series by Greene and Winters on April 29*^, 

April 30*, and May 1*‘, advocating for vouchers in general and special education vouchers 
in particular. See 

http://washingtontimes.eom/article/20080429/EDITORIAL/399369326/1013/editorial , 
http://washingtontimes.com/apps/pbcs.dll/article ?AID=/20080430/EDITORIAL/1 19143777 
and http://washingtontimes.eom/article/20080501/EDITORIAL/670099657 (All retrieved 
May 20, 2008.) 

^ Green, J. P., & Winters, M. (2008). “The Effect of Special Education Vouchers on Public School 
Achievement: Evidence from Elorida's McKay Scholarship Program.” Manhattan Institute, 
Retrieved May 18, 2008, from http://www.manhattan-institute.org/pdf/ 

Effect of Vouchers for SE Students on Public School Achievement 2-19-08.pdf 
(Technical Version) 

^ The Manhattan Institute released two versions of this report — a ‘general release’ report and a more 
detailed version (hereinafter, the ‘technical version’) made available on the Manhattan In- 
stitute website, also entitled, “The Effect of Special Education Vouchers on Public School 
Achievement: Evidence from Elorida's McKay Scholarship Program.” 

Retrieved May 18, 2008, from http://www.manhattan-institute.org/pdf/ 

Effect of Vouchers for SE Students on Public School Achievement 2-19-08.pdf . 

This review is based on the technical version of the report. 

^ Selection bias is defined as bias in estimates due to how samples are selected and are unrelated to 
the actual underlying phenomenon that is being estimated. For instance, in the case of the 
McKay Scholarships, students who receive the vouchers and leave the system are only in- 
cluded in early years of the Manhattan analysis. If these students are different in some way 
from those who stay (which is quite likely) then estimates of special education students’ 
progress are likely to be biased. The direction of that bias is unclear and there is no attempt 
in this analysis to determine directionality. 

^ “Though our ability to evaluate the progress of individual students over time through the use of 
panel-data with individual fixed effects may help to mitigate those sample selection issues 
by accounting for unobserved student heterogeneity, these techniques do not account for 
non-random attrition entirely. Unfortunately there are no variables available in our dataset 
that could serve as a reasonable instrument to account for these sample selection problems, 
and thus we are unable to correct for this bias statistically” (p. 13 of the technical version). 

^ The authors go further in their Washington Times editorial comments, writing, for example, “What 
we know from our study is that rather than harming public schools, vouchers improve the 
education that they provide to their disabled students.” Retrieved May 20, 2008, from 
http://washingtontimes.eom/article/20080429/EDITORIAL/399369326 . This sort of causal 
statement cannot be supported by the analyses reported in the Manhattan study. 

^ Earlier this year, Damian Betebenner wrote a think tank review of one such study. See 

Betebenner, D. (2008, Jan. 15). Review of “Feeling the Florida Heat? How Fow- Performing Schools 
Respond to Voucher and Accountability Pressure. ” Boulder and Tempe: Education and the 
Public Interest Center & Education Policy Research Unit. Retrieved May 20, 2008, from 
http://epicpolicv.org/thinktank/review-feeling-florida-heat-how-low-performing-schools- 
respond-voucher-and-accountability- 

* I emailed the authors for the full tables and even within these full tables it appears that there are 

only four years included in the analysis. 

® A possible reason for the choice of number of schools could have been an extension of a common 
approach used to determine probability of enrolling in private schools: distance to the near- 
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est private school. Why this measure wasn’t employed, but the density measure was em- 
ployed, is unclear. 

Fixed-effects analyses are used when there are multiple observations clustered in some way (such 
as multiple observations for a single individual, or many observations within a single 
school, which is the situation in this analysis). This clustering presents a problem since ob- 
servations clustered in this way violate the assumption in ordinary linear regressions that all 
observations be independent of one another. Fixed-effects provide a way to take into ac- 
count the clustering of observations by looking only at deviations within the clusters around 
the means of the clustered groups. 

In subsequent contacts with the authors, they confirmed that they used the developmental scores in 
their analysis. However, failure to include such an important piece of information in the text 
of the report is a critical oversight and contributes to the lack of clarity throughout the re- 
port. 

Note that this is a discussion of incentives and of potential selection bias scenarios that should be 
considered; it is not an accusation that any public school educators are engaged in such 
“counseling out.” 

The nature and extent of this incentive would depend on the financial and other costs of educating 
a given student or category of students, in addition to the value of the voucher. 



The Think Tank Review Project is made possible by funding from the 
Great Lakes Center for Education Research and Practice. 
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